core: change Interest-combining to play nicer with multiple subscribers
#920
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
|
cc @jeromegn, this should fix the filter leaking you reported in #902. However, we still need to figure out a better strategy for invalidating the callsite cache at the end of a subscriber lifetime, for performance reasons. This change fixes incorrect filtering by ensuring that the current subscriber is always asked whether it will enable a span/event when multiple subscribers disagree on whether or not it should be enabled. However, this means that if we have two subscribers, and one is dropped, we will continue asking the current subscriber for every callsite, even if it would rather return a cached interest, so that we can skip filtering. |
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
yaahc
approved these changes
Aug 12, 2020
Comment on lines
+31
to
+45
| // Iterate over the subscribers in the registry, and — if they are | ||
| // active — register the callsite with them. | ||
| let mut interests = self | ||
| .dispatchers | ||
| .iter() | ||
| .filter_map(|registrar| registrar.try_register(meta)); | ||
|
|
||
| // Use the first subscriber's `Interest` as the base value. | ||
| let interest = if let Some(interest) = interests.next() { | ||
| // Combine all remaining `Interest`s. | ||
| interests.fold(interest, Interest::and) | ||
| } else { | ||
| // If nobody was interested in this thing, just return `never`. | ||
| Interest::never() | ||
| }; |
hawkw
added a commit
that referenced
this pull request
Aug 13, 2020
…bers (#927) ## Motivation Currently, when multiple subscribers are in use, the way that `Interest` values are combined results in surprising behavior. If _any_ subscriber returns `Interest::always` for a callsite, that interest currently takes priority over any other `Interest`s. This means that if two threads have separate subscribers, one of which enables a callsite `always`, and the other `never`, _both_ subscribers will always record that callsite, without the option to disable it. This is not quite right. Instead, we should check whether the current subscriber will enable that callsite in this case. This issue is described in #902. ## Solution This branch changes the rules for combining `Interest`s so that `always` is only returned if _both_ `Interest`s are `always`. If only _one_ is `always`, the combined interest is now `sometimes`, instead. This means that when multiple subscribers exist, `enabled` will always be called on the _current_ subscriber, except when _all_ subscribers have indicated that they are `always` or `never` interested in a callsite. I've added tests that reproduce the issues with leaky filters. Fixing this revealed an additional issue where `tracing-subscriber`'s `EnvFilter` assumes that `enabled` is only called for events, and never for spans, because it always returns either `always` or `never` from `register_callsite` for spans. Therefore, the dynamic span matching directives that might enable a span were never checked in `enabled`. However, under the new interest-combining rules, `enabled` *may* still be called even if a subscriber returns an `always` or `never` interest, if *another* subscriber exists that returned an incompatible interest. This PR fixes that, as well. Depends on #919 This was previously approved by @yaahc as PR #920, but I accidentally merged it into #919 _after_ that branch merged, rather than into master (my bad!). Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently, when multiple subscribers are in use, the way that
Interestvalues are combined results in surprising behavior.
If any subscriber returns
Interest::alwaysfor a callsite, thatinterest currently takes priority over any other
Interests. This meansthat if two threads have separate subscribers, one of which enables a
callsite
always, and the othernever, both subscribers will alwaysrecord that callsite, without the option to disable it. This is not
quite right. Instead, we should check whether the current subscriber
will enable that callsite in this case.
Solution
This branch changes the rules for combining
Interests so thatalwaysis only returned if both
Interests arealways. If only one isalways, the combined interest is nowsometimes, instead. This meansthat when multiple subscribers exist,
enabledwill always be called onthe current subscriber, except when all subscribers have indicated
that they are
alwaysorneverinterested in a callsite.I've added tests that reproduce the issues with leaky filters.
Fixing this revealed an additional issue where
tracing-subscriber'sEnvFilterassumes thatenabledis only called for events, and neverfor spans, because it always returns either
alwaysorneverfromregister_callsitefor spans. Therefore, the dynamic span matchingdirectives that might enable a span were never checked in
enabled.However, under the new interest-combining rules,
enabledmay stillbe called even if a subscriber returns an
alwaysorneverinterest,if another subscriber exists that returned an incompatible interest.
This PR fixes that, as well.
Depends on #919